Denoising in Representation Space via Data-Dependent Regularization for Better Representation

Despite the success of deep learning models, it remains challenging for over-parameterized model to learn good representation under small-sample-size settings. In this paper, motivated by previous work on out-of-distribution (OoD) generalization, we study problem from an OoD perspective identify fundamental factors affecting quality. We formulate a notion “out-of-feature subspace (OoFS) noise” first time, and link OoFS noise in feature extractor performance proving two theorems that demonstrate reducing is beneficial achieving better representation. Moreover, causes prove induced random initialization can be filtered out via L2 regularization. Finally, propose novel data-dependent regularizer acts weights fully connected layer reduce representations, thus implicitly forcing focus informative features rely less back-propagation. Experiments synthetic datasets show our method hard-to-learn features; filter effectively; outperforms GD, AdaGrad, KFAC. Furthermore, experiments benchmark achieves best three tasks among four.